{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IqM-T1RTzY6C"
   },
   "source": [
    "[Unsloth](https://github.com/unslothai/unsloth)是Unsloth AI出品的一个高效的微调工具。它包含闭源和开源两个版本，在这里我们仅关注开源版本的使用。Unsloth重写了模型的内核，尤其是使用triton重写了loss、norm等算子，并手动重写了反向传播机制，使得模型的训练速度更快。同时，Unsloth使用了低精度量化结合LoRA进行微调，进一步降低了显存占用。\n",
    "\n",
    "* Unsloth支持Llama、Mistral、Phi-3、Gemma、Yi、DeepSeek、Qwen、TinyLlama、Vicuna、Open Hermes等\n",
    "* Unsloth支持16bit LoRA或4bit QLoRA。两者都快2倍。\n",
    "* `max_seq_length`可以设置为任何值，因为通过了[kaiokendev的方法](https://kaiokendev.github.io/til)进行自动RoPE缩放。\n",
    "* Unsloth使Gemma-2 9b / 27b **快2倍** 速度运行\n",
    "* 支持自动导出到Ollama\n",
    "\n",
    "要在您自己的计算机上安装Unsloth，请按照Github页面上的安装说明进行操作[这里](https://github.com/unslothai/unsloth?tab=readme-ov-file#-installation-instructions)。\n",
    "\n",
    "笔记本中的功能：\n",
    "1. 使用[FineTome 100K](https://www.modelscope.cn/datasets/AI-ModelScope/FineTome-100k)数据集进行训练。\n",
    "1. 通过`standardize_sharegpt`将ShareGPT转换为标准格式\n",
    "2. 通过`train_on_responses_only`仅在完成/助手上进行训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecutionIndicator": {
     "show": true
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T08:36:19.428819Z",
     "iopub.status.busy": "2024-12-30T08:36:19.428314Z",
     "iopub.status.idle": "2024-12-30T08:36:33.712505Z",
     "shell.execute_reply": "2024-12-30T08:36:33.711914Z",
     "shell.execute_reply.started": "2024-12-30T08:36:19.428798Z"
    },
    "id": "2eSvM9zX_2d3",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://mirrors.aliyun.com/pypi/simple\n",
      "Collecting git+https://github.com/tastelikefeet/unsloth.git@feat/modelscope\n",
      "  Cloning https://github.com/tastelikefeet/unsloth.git (to revision feat/modelscope) to /tmp/pip-req-build-ylwo66va\n",
      "  Running command git clone --filter=blob:none --quiet https://github.com/tastelikefeet/unsloth.git /tmp/pip-req-build-ylwo66va\n",
      "  Running command git checkout -b feat/modelscope --track origin/feat/modelscope\n",
      "  切换到一个新分支 'feat/modelscope'\n",
      "  分支 'feat/modelscope' 设置为跟踪来自 'origin' 的远程分支 'feat/modelscope'。\n",
      "  Resolved https://github.com/tastelikefeet/unsloth.git to commit 7146bc2f759a9a2f300f4009b1d3ab671bb8dbb5\n",
      "  Installing build dependencies ... \u001b[?25ldone\n",
      "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n",
      "\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n",
      "\u001b[?25h\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.3.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# !pip install unsloth\n",
    "# !pip uninstall unsloth -y && pip install --upgrade --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.git\n",
    "# 截止到当前文件编写的时候，modelscope的pr目前没有合并进主分支，请使用这个命令安装unsloth：\n",
    "!pip install git+https://github.com/tastelikefeet/unsloth.git@feat/modelscope"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面我们拉起模型。注意unsloth项目组在modelscope社区上提供了许多模型，如果您需要使用的模型在魔搭上不存在，请考虑告诉我们，或者直接上传一个。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecutionIndicator": {
     "show": true
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 301,
     "referenced_widgets": [
      "70b0793d26c14afbbe8af319d38dbb07",
      "20e75852859845f697760003945b952f",
      "f0c7c3402e6549148ac293dcf01ae8da",
      "b1c8acb40b924d8f854408252c1e061c",
      "aa28da40d8074e89ad32b5cb5ad04fa6",
      "4a91a45aba634535a53a49a873b25dfe",
      "d773266178564b6cb739d756ff8de459",
      "cff4f20460a442839c26bfa468b35dee",
      "683dd21b9eef4bb4a0c9bc15c984703a",
      "f0377d014b2e4e89b9932750f04a6a3c",
      "0d384a63bb5f4b7384ac5f20d1a399b0",
      "4606a430f0214a0bb524a06ab78af1a9",
      "700b2fc6799c41f4984e07c36502f2c9",
      "be744decce12471388b689a085576c7f",
      "e8efa5951c4b4f3e9922e029c57a5ab3",
      "afa86659651d48a7b4a3043a0909322f",
      "9edc5f3740e54d7ab6fd38f270f4a40e",
      "2aea4e74b6c4449ea24812d11675daee",
      "252c3d25fe3341379e5fc31d30dee7bc",
      "672262ede93c413cbc219ca4fec5b98a",
      "ea0f4a924ce14b97a53ba2b045ba1b1d",
      "757a4328afef4293b6d31ce9b90eec79",
      "18c4cb423f8744afa2b85ee361dcc121",
      "76fb012b818f4bfb88c103262fb9c20e",
      "8158ce20191d40fba140ed2c2f4ed859",
      "595694711ab148e8bbf9824896c10468",
      "4b355eddebb548a7b60d28c12403ebfd",
      "0faf73ed502649c4855adca930e3c8ff",
      "b0670b92e79744c5ba11501bafdc101b",
      "3ea24021851a4341b3cb30d4ff0d495f",
      "7028f81557e54d0d8df4cc1a0f85b12c",
      "0b8cf23562ae4e428ab4f00979e7ce07",
      "b774631435d4450da64f0a2b40cbf3aa",
      "8578be7c3f4044d3bfe0eae94fe2c21a",
      "4f0aaa3ce8ca43fdaea4cd4f7b595218",
      "8611f4c6d12641dd960377346daba5d1",
      "3601b111b63f401cb5c75ba08ae74b61",
      "a5bde1dcf7f745faa140cb6728c415fc",
      "6d163ef8e07b4a1cba351ec44175efd5",
      "9a4ea589183346dab240125cd38ba0f9",
      "d8f739452aa5410aa5eba87b2760c2af",
      "dc098c4a448243ddb82ab769dc261171",
      "d8c3cd1d08b747b690a59b7793aaf742",
      "95a6cb0e04984f3c96bbe9ee9c5fa287",
      "254473850de1471e9584a9c03962bd6f",
      "56e59a69f84e487f91e263313f5d2b72",
      "524b13439d53402eb35388606058c288",
      "fc7d250163bb470fa19503356dfe9e81",
      "9dc2697fe6984649b0ac36018f6904c9",
      "3ebff50e3c0b4cb4b99c93adde35b712",
      "35aee367658940b9b2f6884ea87b3e4b",
      "f7f53b3a53eb4b8e9fa47cd233f4df7f",
      "73efb213f4a84996a353495b912393d9",
      "8b09015cc22240cc8e736f7050c0f37c",
      "c7907c51d16a4aeb87974c36b3b9e7c8"
     ]
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:01:40.310562Z",
     "iopub.status.busy": "2024-12-30T09:01:40.310304Z",
     "iopub.status.idle": "2024-12-30T09:01:48.648813Z",
     "shell.execute_reply": "2024-12-30T09:01:48.648168Z",
     "shell.execute_reply.started": "2024-12-30T09:01:40.310540Z"
    },
    "id": "QmUBVEnvCDJv",
    "outputId": "cb455c4b-2327-4a15-9078-de2fef37d9bd",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unsloth: Your Flash Attention 2 installation seems to be broken?\n",
      "A possible explanation is you have a new CUDA version which isn't\n",
      "yet compatible with FA2? Please file a ticket to Unsloth or FA2.\n",
      "We shall now use Xformers instead, which does not have any performance hits!\n",
      "We found this negligible impact by benchmarking on 1x A100.\n",
      "🦥 Unsloth Zoo will now patch everything to make training faster!\n",
      "[2024-12-30 17:01:45,179] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)\n",
      "use modelscope\n",
      "Downloading Model to directory: /mnt/workspace/.cache/modelscope/hub/unsloth/llama-3.2-3b-instruct-bnb-4bit\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-12-30 17:01:46,115 - modelscope - INFO - Target directory already exists, skipping creation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==((====))==  Unsloth 2024.12.9: Fast Llama patching. Transformers: 4.47.1.\n",
      "   \\\\   /|    GPU: NVIDIA A100-SXM4-80GB. Max memory: 79.347 GB. Platform: Linux.\n",
      "O^O/ \\_/ \\    Torch: 2.4.0+cu121. CUDA: 8.0. CUDA Toolkit: 12.1. Triton: 3.0.0\n",
      "\\        /    Bfloat16 = TRUE. FA [Xformers = 0.0.27.post2. FA2 = False]\n",
      " \"-____-\"     Free Apache license: http://github.com/unslothai/unsloth\n",
      "Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "# 对于国内用户，魔搭社区提供了快速下载模型和数据集的方法，只需要简单引入一个环境变量:\n",
    "os.environ['UNSLOTH_USE_MODELSCOPE'] = 'true'\n",
    "from unsloth import FastLanguageModel\n",
    "import torch\n",
    "max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally!\n",
    "dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+\n",
    "load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.\n",
    "\n",
    "# 4bit pre quantized models we support for 4x faster downloading + no OOMs.\n",
    "fourbit_models = [\n",
    "    \"unsloth/Meta-Llama-3.1-8B-bnb-4bit\",      # Llama-3.1 2x faster\n",
    "    \"unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit\",\n",
    "    \"unsloth/Meta-Llama-3.1-70B-bnb-4bit\",\n",
    "    \"unsloth/Meta-Llama-3.1-405B-bnb-4bit\",    # 4bit for 405b!\n",
    "    \"unsloth/Mistral-Small-Instruct-2409\",     # Mistral 22b 2x faster!\n",
    "    \"unsloth/mistral-7b-instruct-v0.3-bnb-4bit\",\n",
    "    \"unsloth/Phi-3.5-mini-instruct\",           # Phi-3.5 2x faster!\n",
    "    \"unsloth/Phi-3-medium-4k-instruct\",\n",
    "    \"unsloth/gemma-2-9b-bnb-4bit\",\n",
    "    \"unsloth/gemma-2-27b-bnb-4bit\",            # Gemma 2x faster!\n",
    "\n",
    "    \"unsloth/Llama-3.2-1B-bnb-4bit\",           # NEW! Llama 3.2 models\n",
    "    \"unsloth/Llama-3.2-1B-Instruct-bnb-4bit\",\n",
    "    \"unsloth/Llama-3.2-3B-bnb-4bit\",\n",
    "    \"unsloth/Llama-3.2-3B-Instruct-bnb-4bit\",\n",
    "\n",
    "    \"unsloth/Llama-3.3-70B-Instruct-bnb-4bit\" # NEW! Llama 3.3 70B!\n",
    "]\n",
    "\n",
    "model, tokenizer = FastLanguageModel.from_pretrained(\n",
    "    model_name = \"unsloth/Llama-3.2-3B-Instruct\", # or choose \"unsloth/Llama-3.2-1B-Instruct\"\n",
    "    max_seq_length = max_seq_length,\n",
    "    dtype = dtype,\n",
    "    load_in_4bit = load_in_4bit,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SXd9bTZd1aaL"
   },
   "source": [
    "现在添加LoRA适配器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:02:18.475150Z",
     "iopub.status.busy": "2024-12-30T09:02:18.474539Z",
     "iopub.status.idle": "2024-12-30T09:02:22.128940Z",
     "shell.execute_reply": "2024-12-30T09:02:22.128310Z",
     "shell.execute_reply.started": "2024-12-30T09:02:18.475119Z"
    },
    "id": "6bZsfBuZDeCL",
    "outputId": "acc0f9f5-59a6-46fe-d5bb-cd09965bb8c9",
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Unsloth 2024.12.9 patched 28 layers with 28 QKV layers, 28 O layers and 28 MLP layers.\n"
     ]
    }
   ],
   "source": [
    "model = FastLanguageModel.get_peft_model(\n",
    "    model,\n",
    "    r = 16, # 选择任何大于0的数字！建议8, 16, 32, 64, 128\n",
    "    target_modules = [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n",
    "                      \"gate_proj\", \"up_proj\", \"down_proj\",],\n",
    "    lora_alpha = 16,\n",
    "    lora_dropout = 0, # 支持任何值，但=0是优化的\n",
    "    bias = \"none\",    # 支持任何值，但=\"none\"是优化的\n",
    "    use_gradient_checkpointing = \"unsloth\", # 对于非常长的上下文使用True或\"unsloth\"\n",
    "    random_state = 3407,\n",
    "    use_rslora = False,  # 我们支持秩稳定的LoRA\n",
    "    loftq_config = None, # 以及LoftQ\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vITh0KVJ10qX"
   },
   "source": [
    "<a name=\"Data\"></a>\n",
    "### 数据准备\n",
    "我们现在使用 `Llama-3.1` 格式进行对话风格的微调。我们使用 [FineTome-100k](https://www.modelscope.cn/datasets/AI-ModelScope/FineTome-100k) 数据集，采用 ShareGPT 风格。但是我们将其转换为多轮格式 `(\"role\", \"content\")`，而不是 `(\"from\", \"value\")`。Llama-3 以如下方式呈现多轮对话：\n",
    "\n",
    "```\n",
    "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n",
    "\n",
    "Hello!<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n",
    "\n",
    "Hey there! How are you?<|eot_id|><|start_header_id|>user<|end_header_id|>\n",
    "\n",
    "I'm great thanks!<|eot_id|>\n",
    "```\n",
    "\n",
    "我们使用 `get_chat_template` 函数来获取正确的聊天模板。我们支持 `zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, phi3, llama3` 等。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecutionIndicator": {
     "show": true
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 113,
     "referenced_widgets": [
      "39bf1c29894f43acb6d2919e64a4fd28",
      "007a35a241b346ec9a5cdd6f3e4ddd27",
      "969a119573f942b29951ae2933e61cde",
      "b8c4d378ea0e4bcd9f572a191a7c136f",
      "7d37dd0e06724b53b4f31cc0a4321b0d",
      "4083b2ef8e6348e18b69d116508b46ff",
      "9555be409a2c4a97b18d4978ed13d35f",
      "5628ed38f304438faf5442b29a9511d6",
      "6e0fe945001140b3959e617a2f55c353",
      "0c30ded692064dc7bf36a93897f2b68f",
      "8c5ad85b4da14b239340ac95244d8ed4",
      "39684b70f2ff48cab454617c721f7777",
      "e8445e90b1054aacbecf198c7979a0b6",
      "d1cc50fb6d5849888af5d765dc51ab62",
      "2b359412d4914aa38a6e21284c12ecbc",
      "a4ceb6dbc8de4fa798ee39d28e5ebc40",
      "d6ab4d4143ff49bcae30be1bc2d76762",
      "904e7bac43bd4333b321cacfed5dcb60",
      "2bb75539976c49ed805c4ff6c58fb1d2",
      "45bc9d882a8f4a7e813245b1590d4427",
      "ddee625828cb4c22927aa73a02cd2dd9",
      "fd46f381983f49179de05497c171c805",
      "785d9147f4a341afafc5c5743892df16",
      "5e9825466cd2481b92cfe89f33b11fe3",
      "bfbb37b6f4b247b5bf5aaf7e1d80bcf9",
      "2a6ca29a76ff430d86213f910858db5b",
      "92d981a21b204f6c8b52e3caa16d1784",
      "c685f29a5d2c461ca3dda867bab6df50",
      "e2f16d56b21c4ff2918872d70e5ca847",
      "0bfbfe620ff446a0a47f7d5de7c88975",
      "5c9ee920068a47d89dbf5cbdd9e848a3",
      "95249b8fb6a84054a01f22c5f73f207b",
      "2ed2b017b9a24f36a4222c5c27753991"
     ]
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:03:34.106887Z",
     "iopub.status.busy": "2024-12-30T09:03:34.106690Z",
     "iopub.status.idle": "2024-12-30T09:03:50.893660Z",
     "shell.execute_reply": "2024-12-30T09:03:50.892954Z",
     "shell.execute_reply.started": "2024-12-30T09:03:34.106870Z"
    },
    "id": "LjY75GoYUCB8",
    "outputId": "94095b01-dac6-4f9c-cbc3-ca78e007ba12",
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading [README.md]: 100%|██████████| 982/982 [00:00<00:00, 6.22MB/s]\n",
      "2024-12-30 17:03:36,084 - modelscope - INFO - storing https://www.modelscope.cn/api/v1/datasets/AI-ModelScope/FineTome-100k/repo?Source=SDK&Revision=master&FilePath=README.md&View=False in cache at /mnt/workspace/.cache/modelscope/hub/datasets/ad10352f0b6842676e6d5943f385c7bb2b07c1abd592c2aa260e9af638ecea41\n",
      "2024-12-30 17:03:36,091 - modelscope - INFO - creating metadata file for /mnt/workspace/.cache/modelscope/hub/datasets/ad10352f0b6842676e6d5943f385c7bb2b07c1abd592c2aa260e9af638ecea41\n",
      "Downloading data: 117MB [00:09, 11.8MB/s] \n",
      "2024-12-30 17:03:48,427 - modelscope - INFO - storing https://www.modelscope.cn/api/v1/datasets/AI-ModelScope/FineTome-100k/repo?Source=SDK&Revision=master&FilePath=data%2Ftrain-00000-of-00001.parquet in cache at /mnt/workspace/.cache/modelscope/hub/datasets/downloads/27484246eca7151cf2513ef6f4a6be4a7e7520bbcdd1b99b2cfd61c4164a7cd6\n",
      "2024-12-30 17:03:48,436 - modelscope - INFO - creating metadata file for /mnt/workspace/.cache/modelscope/hub/datasets/downloads/27484246eca7151cf2513ef6f4a6be4a7e7520bbcdd1b99b2cfd61c4164a7cd6\n",
      "Generating train split: 100%|██████████| 100000/100000 [00:01<00:00, 54443.18 examples/s]\n"
     ]
    }
   ],
   "source": [
    "from unsloth.chat_templates import get_chat_template\n",
    "\n",
    "tokenizer = get_chat_template(\n",
    "    tokenizer,\n",
    "    chat_template = \"llama-3.1\",\n",
    ")\n",
    "\n",
    "def formatting_prompts_func(examples):\n",
    "    convos = examples[\"conversations\"]\n",
    "    texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos]\n",
    "    return { \"text\" : texts, }\n",
    "pass\n",
    "\n",
    "from modelscope import MsDataset\n",
    "dataset = MsDataset.load(\"AI-ModelScope/FineTome-100k\", split = \"train\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K9CBpiISFa6C"
   },
   "source": [
    "现在使用 `standardize_sharegpt` 将 ShareGPT 风格的数据集转换为Conversations通用格式。这将数据集从如下格式:\n",
    "```\n",
    "{\"from\": \"system\", \"value\": \"You are an assistant\"}\n",
    "{\"from\": \"human\", \"value\": \"What is 2+2?\"}\n",
    "{\"from\": \"gpt\", \"value\": \"It's 4.\"}\n",
    "```\n",
    "转换为\n",
    "```\n",
    "{\"role\": \"system\", \"content\": \"You are an assistant\"}\n",
    "{\"role\": \"user\", \"content\": \"What is 2+2?\"}\n",
    "{\"role\": \"assistant\", \"content\": \"It's 4.\"}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 81,
     "referenced_widgets": [
      "dd9e90f2c16541e8a72c6771c4685b9a",
      "a326b2e89f1c46f28cd166afc7490e2b",
      "eb855a0fcb554a8eb245351b3593623d",
      "bd71b6cb29e147ab9b10d1b85908c413",
      "b1b0a4e3f00043b0a0eb7a053815a4a5",
      "58ce4633471c438db6e103a1ca3806a0",
      "cf1b769b7a744b5f8bccf6798566582f",
      "1c0c2835705f41089de4caea98127c04",
      "e2d886444f0047fa9e2245b9773ced9e",
      "c03b9410af384397849ef63b62f2c689",
      "098bd8ace574423da763eb0eae1d3bb6",
      "d08e764aa8b94e7f9e1c727b53980abe",
      "e62f6eb58a744d38b837e47d8a16db67",
      "bcf8e36d938a4d959c31ea4ff3c8d4cf",
      "ae2464c1cbc442a383de7577d2986116",
      "9a8f1b8079fe478ebf0b16096cb224f5",
      "e4bf3f8e63bb4c01bbe821d438445d91",
      "d7e0024b98a94a9fa12dc4154ff2b2fc",
      "cc0bd79ca9e847fba88aafe2d612ffe4",
      "76e2e47c93e541ff820bcbab9264381d",
      "4b41aa65c6894e918b04709f8e9270d2",
      "cdae06929214464ea25e343f17b4a843"
     ]
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:04:23.348211Z",
     "iopub.status.busy": "2024-12-30T09:04:23.347241Z",
     "iopub.status.idle": "2024-12-30T09:04:38.154900Z",
     "shell.execute_reply": "2024-12-30T09:04:38.154285Z",
     "shell.execute_reply.started": "2024-12-30T09:04:23.348182Z"
    },
    "id": "oPXzJZzHEgXe",
    "outputId": "dd1c72fa-39ea-48a2-9ed2-c263a4549b91",
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Standardizing format: 100%|██████████| 100000/100000 [00:04<00:00, 20475.55 examples/s]\n",
      "Map: 100%|██████████| 100000/100000 [00:09<00:00, 10242.12 examples/s]\n"
     ]
    }
   ],
   "source": [
    "from unsloth.chat_templates import standardize_sharegpt\n",
    "dataset = standardize_sharegpt(dataset)\n",
    "dataset = dataset.map(formatting_prompts_func, batched = True,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ndDUB23CGAC5"
   },
   "source": [
    "我们来看一下第0项的对话结构："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:04:55.928686Z",
     "iopub.status.busy": "2024-12-30T09:04:55.928172Z",
     "iopub.status.idle": "2024-12-30T09:04:55.940707Z",
     "shell.execute_reply": "2024-12-30T09:04:55.940258Z",
     "shell.execute_reply.started": "2024-12-30T09:04:55.928662Z"
    },
    "id": "gGFzmplrEy9I",
    "outputId": "9f3f66fc-8649-40c8-829c-db3f11f88728",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'content': 'Explain what boolean operators are, what they do, and provide examples of how they can be used in programming. Additionally, describe the concept of operator precedence and provide examples of how it affects the evaluation of boolean expressions. Discuss the difference between short-circuit evaluation and normal evaluation in boolean expressions and demonstrate their usage in code. \\n\\nFurthermore, add the requirement that the code must be written in a language that does not support short-circuit evaluation natively, forcing the test taker to implement their own logic for short-circuit evaluation.\\n\\nFinally, delve into the concept of truthiness and falsiness in programming languages, explaining how it affects the evaluation of boolean expressions. Add the constraint that the test taker must write code that handles cases where truthiness and falsiness are implemented differently across different programming languages.',\n",
       "  'role': 'user'},\n",
       " {'content': 'Boolean operators are logical operators used in programming to manipulate boolean values. They operate on one or more boolean operands and return a boolean result. The three main boolean operators are \"AND\" (&&), \"OR\" (||), and \"NOT\" (!).\\n\\nThe \"AND\" operator returns true if both of its operands are true, and false otherwise. For example:\\n\\n```python\\nx = 5\\ny = 10\\nresult = (x > 0) and (y < 20)  # This expression evaluates to True\\n```\\n\\nThe \"OR\" operator returns true if at least one of its operands is true, and false otherwise. For example:\\n\\n```python\\nx = 5\\ny = 10\\nresult = (x > 0) or (y < 20)  # This expression evaluates to True\\n```\\n\\nThe \"NOT\" operator negates the boolean value of its operand. It returns true if the operand is false, and false if the operand is true. For example:\\n\\n```python\\nx = 5\\nresult = not (x > 10)  # This expression evaluates to True\\n```\\n\\nOperator precedence refers to the order in which operators are evaluated in an expression. It ensures that expressions are evaluated correctly. In most programming languages, logical AND has higher precedence than logical OR. For example:\\n\\n```python\\nresult = True or False and False  # This expression is evaluated as (True or (False and False)), which is True\\n```\\n\\nShort-circuit evaluation is a behavior where the second operand of a logical operator is not evaluated if the result can be determined based on the value of the first operand. In short-circuit evaluation, if the first operand of an \"AND\" operator is false, the second operand is not evaluated because the result will always be false. Similarly, if the first operand of an \"OR\" operator is true, the second operand is not evaluated because the result will always be true.\\n\\nIn programming languages that support short-circuit evaluation natively, you can use it to improve performance or avoid errors. For example:\\n\\n```python\\nif x != 0 and (y / x) > 10:\\n    # Perform some operation\\n```\\n\\nIn languages without native short-circuit evaluation, you can implement your own logic to achieve the same behavior. Here\\'s an example in pseudocode:\\n\\n```\\nif x != 0 {\\n    if (y / x) > 10 {\\n        // Perform some operation\\n    }\\n}\\n```\\n\\nTruthiness and falsiness refer to how non-boolean values are evaluated in boolean contexts. In many programming languages, non-zero numbers and non-empty strings are considered truthy, while zero, empty strings, and null/None values are considered falsy.\\n\\nWhen evaluating boolean expressions, truthiness and falsiness come into play. For example:\\n\\n```python\\nx = 5\\nresult = x  # The value of x is truthy, so result is also truthy\\n```\\n\\nTo handle cases where truthiness and falsiness are implemented differently across programming languages, you can explicitly check the desired condition. For example:\\n\\n```python\\nx = 5\\nresult = bool(x)  # Explicitly converting x to a boolean value\\n```\\n\\nThis ensures that the result is always a boolean value, regardless of the language\\'s truthiness and falsiness rules.',\n",
       "  'role': 'assistant'}]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[0][\"conversations\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "idAEIeSQ3xdS"
   },
   "source": [
    "<a name=\"Train\"></a>\n",
    "### 训练模型\n",
    "作为例子，我们使用 Huggingface TRL 的 `SFTTrainer`进行训练："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67,
     "referenced_widgets": [
      "3ffe42931dcf4a69972f4d50ee4dd3dd",
      "ee9dcec2d5c44fd883f16c06b9f76264",
      "982b6b94642d49fa85fab6ad621392fe",
      "42990f347a8c42f7b510e2d17c7d3c6e",
      "3cd95b7c5e2f4c6883333045db11c6d6",
      "5b34a4e8fc7747e78b49ad5bf67a6580",
      "23907906314743938db4e484c15480cc",
      "378176d2f0c9466d8762a584edf4217d",
      "e221482cbe95465191212d85d539938c",
      "74dc78a38e30465a96d2c8a22a27b127",
      "c6b4759ce826421081508270cb30334b"
     ]
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:05:04.680858Z",
     "iopub.status.busy": "2024-12-30T09:05:04.680307Z",
     "iopub.status.idle": "2024-12-30T09:06:09.429020Z",
     "shell.execute_reply": "2024-12-30T09:06:09.428350Z",
     "shell.execute_reply.started": "2024-12-30T09:05:04.680835Z"
    },
    "id": "95_Nn-89DhsL",
    "outputId": "97211c96-b8e2-4b35-8691-892550ee0e7a",
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Map (num_proc=2): 100%|██████████| 100000/100000 [01:04<00:00, 1551.82 examples/s]\n"
     ]
    }
   ],
   "source": [
    "from trl import SFTTrainer\n",
    "from transformers import TrainingArguments, DataCollatorForSeq2Seq\n",
    "from unsloth import is_bfloat16_supported\n",
    "\n",
    "trainer = SFTTrainer(\n",
    "    model = model,\n",
    "    tokenizer = tokenizer,\n",
    "    train_dataset = dataset,\n",
    "    dataset_text_field = \"text\",\n",
    "    max_seq_length = max_seq_length,\n",
    "    data_collator = DataCollatorForSeq2Seq(tokenizer = tokenizer),\n",
    "    dataset_num_proc = 2,\n",
    "    packing = False, # 对于短序列可以使训练速度提高5倍。\n",
    "    args = TrainingArguments(\n",
    "        per_device_train_batch_size = 2,\n",
    "        gradient_accumulation_steps = 4,\n",
    "        warmup_steps = 5,\n",
    "        # num_train_epochs = 1, # 设置为1次完整的训练运行。\n",
    "        max_steps = 60,\n",
    "        learning_rate = 2e-4,\n",
    "        fp16 = not is_bfloat16_supported(),\n",
    "        bf16 = is_bfloat16_supported(),\n",
    "        logging_steps = 1,\n",
    "        optim = \"adamw_8bit\",\n",
    "        weight_decay = 0.01,\n",
    "        lr_scheduler_type = \"linear\",\n",
    "        seed = 3407,\n",
    "        output_dir = \"outputs\",\n",
    "        report_to = \"none\", # 用于WandB等\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "C_sGp5XlG6dq"
   },
   "source": [
    "使用Unsloth的`train_on_completions`方法，仅对助手的输出进行训练，并忽略用户输入的损失。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecutionIndicator": {
     "show": true
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49,
     "referenced_widgets": [
      "6064feeea79040409e18a1e2a289b09a",
      "bb241a26ca4d4d7186ba46cda1f8a802",
      "c9abb42da1734388a7d2f1a06832ecc6",
      "7c3a37494e5848b9994b37a4c8bac132",
      "c668ae4c7d174f2dad3fb837ff873e57",
      "dd30f3ead6394317be5a72aa890adfb9",
      "1e4ea03959b3496f8e75cc3588cf347c",
      "d356b597dda14c7ab023403ee6959cf8",
      "870ff8f17c7b47ec8d49cac84216b04c",
      "d5cfa138483f4007b2a95be833043235",
      "6d52daf29c90402a9762acdde765713f"
     ]
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:07:56.880355Z",
     "iopub.status.busy": "2024-12-30T09:07:56.879474Z",
     "iopub.status.idle": "2024-12-30T09:08:30.611780Z",
     "shell.execute_reply": "2024-12-30T09:08:30.611262Z",
     "shell.execute_reply.started": "2024-12-30T09:07:56.880326Z"
    },
    "id": "juQiExuBG5Bt",
    "outputId": "dca88e73-ac69-4199-9c83-cb6300e8ce9a",
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Map: 100%|██████████| 100000/100000 [00:33<00:00, 2965.92 examples/s]\n"
     ]
    }
   ],
   "source": [
    "from unsloth.chat_templates import train_on_responses_only\n",
    "trainer = train_on_responses_only(\n",
    "    trainer,\n",
    "    instruction_part = \"<|start_header_id|>user<|end_header_id|>\\n\\n\",\n",
    "    response_part = \"<|start_header_id|>assistant<|end_header_id|>\\n\\n\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Dv1NBUozV78l"
   },
   "source": [
    "验证掩码是否实际完成："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 159
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:09:07.324860Z",
     "iopub.status.busy": "2024-12-30T09:09:07.324551Z",
     "iopub.status.idle": "2024-12-30T09:09:07.340782Z",
     "shell.execute_reply": "2024-12-30T09:09:07.340222Z",
     "shell.execute_reply.started": "2024-12-30T09:09:07.324839Z"
    },
    "id": "LtsMVtlkUhja",
    "outputId": "84735ea5-8489-4a34-f501-afe91901d542",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nCutting Knowledge Date: December 2023\\nToday Date: 26 July 2024\\n\\n<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\nExplain what boolean operators are, what they do, and provide examples of how they can be used in programming. Additionally, describe the concept of operator precedence and provide examples of how it affects the evaluation of boolean expressions. Discuss the difference between short-circuit evaluation and normal evaluation in boolean expressions and demonstrate their usage in code. \\n\\nFurthermore, add the requirement that the code must be written in a language that does not support short-circuit evaluation natively, forcing the test taker to implement their own logic for short-circuit evaluation.\\n\\nFinally, delve into the concept of truthiness and falsiness in programming languages, explaining how it affects the evaluation of boolean expressions. Add the constraint that the test taker must write code that handles cases where truthiness and falsiness are implemented differently across different programming languages.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\nBoolean operators are logical operators used in programming to manipulate boolean values. They operate on one or more boolean operands and return a boolean result. The three main boolean operators are \"AND\" (&&), \"OR\" (||), and \"NOT\" (!).\\n\\nThe \"AND\" operator returns true if both of its operands are true, and false otherwise. For example:\\n\\n```python\\nx = 5\\ny = 10\\nresult = (x > 0) and (y < 20)  # This expression evaluates to True\\n```\\n\\nThe \"OR\" operator returns true if at least one of its operands is true, and false otherwise. For example:\\n\\n```python\\nx = 5\\ny = 10\\nresult = (x > 0) or (y < 20)  # This expression evaluates to True\\n```\\n\\nThe \"NOT\" operator negates the boolean value of its operand. It returns true if the operand is false, and false if the operand is true. For example:\\n\\n```python\\nx = 5\\nresult = not (x > 10)  # This expression evaluates to True\\n```\\n\\nOperator precedence refers to the order in which operators are evaluated in an expression. It ensures that expressions are evaluated correctly. In most programming languages, logical AND has higher precedence than logical OR. For example:\\n\\n```python\\nresult = True or False and False  # This expression is evaluated as (True or (False and False)), which is True\\n```\\n\\nShort-circuit evaluation is a behavior where the second operand of a logical operator is not evaluated if the result can be determined based on the value of the first operand. In short-circuit evaluation, if the first operand of an \"AND\" operator is false, the second operand is not evaluated because the result will always be false. Similarly, if the first operand of an \"OR\" operator is true, the second operand is not evaluated because the result will always be true.\\n\\nIn programming languages that support short-circuit evaluation natively, you can use it to improve performance or avoid errors. For example:\\n\\n```python\\nif x!= 0 and (y / x) > 10:\\n    # Perform some operation\\n```\\n\\nIn languages without native short-circuit evaluation, you can implement your own logic to achieve the same behavior. Here\\'s an example in pseudocode:\\n\\n```\\nif x!= 0 {\\n    if (y / x) > 10 {\\n        // Perform some operation\\n    }\\n}\\n```\\n\\nTruthiness and falsiness refer to how non-boolean values are evaluated in boolean contexts. In many programming languages, non-zero numbers and non-empty strings are considered truthy, while zero, empty strings, and null/None values are considered falsy.\\n\\nWhen evaluating boolean expressions, truthiness and falsiness come into play. For example:\\n\\n```python\\nx = 5\\nresult = x  # The value of x is truthy, so result is also truthy\\n```\\n\\nTo handle cases where truthiness and falsiness are implemented differently across programming languages, you can explicitly check the desired condition. For example:\\n\\n```python\\nx = 5\\nresult = bool(x)  # Explicitly converting x to a boolean value\\n```\\n\\nThis ensures that the result is always a boolean value, regardless of the language\\'s truthiness and falsiness rules.<|eot_id|>'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(trainer.train_dataset[0][\"input_ids\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecutionIndicator": {
     "show": true
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 106
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:09:09.841937Z",
     "iopub.status.busy": "2024-12-30T09:09:09.841090Z",
     "iopub.status.idle": "2024-12-30T09:09:09.848999Z",
     "shell.execute_reply": "2024-12-30T09:09:09.848248Z",
     "shell.execute_reply.started": "2024-12-30T09:09:09.841909Z"
    },
    "id": "_rD6fl8EUxnG",
    "outputId": "7b0d0ab4-06c3-4f2c-bb94-0ec853a4d0cc",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 271, 7035, 20197, 527, 20406, 20197, 1511, 304, 15840, 311, 37735, 2777, 2819, 13, 2435, 14816, 389, 832, 477, 810, 2777, 55610, 323, 471, 264, 2777, 1121, 13, 578, 2380, 1925, 2777, 20197, 527, 330, 4064, 1, 320, 7827, 705, 330, 878, 1, 320, 8651, 705, 323, 330, 14394, 1, 1533, 3677, 791, 330, 4064, 1, 5793, 4780, 837, 422, 2225, 315, 1202, 55610, 527, 837, 11, 323, 905, 6062, 13, 1789, 3187, 1473, 74694, 12958, 198, 87, 284, 220, 20, 198, 88, 284, 220, 605, 198, 1407, 284, 320, 87, 871, 220, 15, 8, 323, 320, 88, 366, 220, 508, 8, 220, 674, 1115, 7645, 67349, 311, 3082, 198, 14196, 19884, 791, 330, 878, 1, 5793, 4780, 837, 422, 520, 3325, 832, 315, 1202, 55610, 374, 837, 11, 323, 905, 6062, 13, 1789, 3187, 1473, 74694, 12958, 198, 87, 284, 220, 20, 198, 88, 284, 220, 605, 198, 1407, 284, 320, 87, 871, 220, 15, 8, 477, 320, 88, 366, 220, 508, 8, 220, 674, 1115, 7645, 67349, 311, 3082, 198, 14196, 19884, 791, 330, 14394, 1, 5793, 4277, 988, 279, 2777, 907, 315, 1202, 28312, 13, 1102, 4780, 837, 422, 279, 28312, 374, 905, 11, 323, 905, 422, 279, 28312, 374, 837, 13, 1789, 3187, 1473, 74694, 12958, 198, 87, 284, 220, 20, 198, 1407, 284, 539, 320, 87, 871, 220, 605, 8, 220, 674, 1115, 7645, 67349, 311, 3082, 198, 14196, 19884, 18968, 54156, 19813, 311, 279, 2015, 304, 902, 20197, 527, 26126, 304, 459, 7645, 13, 1102, 26420, 430, 24282, 527, 26126, 12722, 13, 763, 1455, 15840, 15823, 11, 20406, 3651, 706, 5190, 54156, 1109, 20406, 2794, 13, 1789, 3187, 1473, 74694, 12958, 198, 1407, 284, 3082, 477, 3641, 323, 3641, 220, 674, 1115, 7645, 374, 26126, 439, 320, 2575, 477, 320, 4139, 323, 3641, 5850, 902, 374, 3082, 198, 14196, 19884, 12755, 1824, 38368, 16865, 374, 264, 7865, 1405, 279, 2132, 28312, 315, 264, 20406, 5793, 374, 539, 26126, 422, 279, 1121, 649, 387, 11075, 3196, 389, 279, 907, 315, 279, 1176, 28312, 13, 763, 2875, 1824, 38368, 16865, 11, 422, 279, 1176, 28312, 315, 459, 330, 4064, 1, 5793, 374, 905, 11, 279, 2132, 28312, 374, 539, 26126, 1606, 279, 1121, 690, 2744, 387, 905, 13, 35339, 11, 422, 279, 1176, 28312, 315, 459, 330, 878, 1, 5793, 374, 837, 11, 279, 2132, 28312, 374, 539, 26126, 1606, 279, 1121, 690, 2744, 387, 837, 382, 644, 15840, 15823, 430, 1862, 2875, 1824, 38368, 16865, 308, 8046, 11, 499, 649, 1005, 433, 311, 7417, 5178, 477, 5766, 6103, 13, 1789, 3187, 1473, 74694, 12958, 198, 333, 865, 976, 220, 15, 323, 320, 88, 611, 865, 8, 871, 220, 605, 512, 262, 674, 26050, 1063, 5784, 198, 14196, 19884, 644, 15823, 2085, 10068, 2875, 1824, 38368, 16865, 11, 499, 649, 4305, 701, 1866, 12496, 311, 11322, 279, 1890, 7865, 13, 5810, 596, 459, 3187, 304, 51743, 44788, 1473, 14196, 4077, 333, 865, 976, 220, 15, 341, 262, 422, 320, 88, 611, 865, 8, 871, 220, 605, 341, 286, 443, 26050, 1063, 5784, 198, 262, 457, 534, 14196, 19884, 25025, 1918, 323, 33032, 1918, 8464, 311, 1268, 2536, 12, 6245, 2819, 527, 26126, 304, 2777, 38697, 13, 763, 1690, 15840, 15823, 11, 2536, 38029, 5219, 323, 2536, 40533, 9246, 527, 6646, 8206, 88, 11, 1418, 7315, 11, 4384, 9246, 11, 323, 854, 14, 4155, 2819, 527, 6646, 33032, 88, 382, 4599, 38663, 2777, 24282, 11, 8206, 1918, 323, 33032, 1918, 2586, 1139, 1514, 13, 1789, 3187, 1473, 74694, 12958, 198, 87, 284, 220, 20, 198, 1407, 284, 865, 220, 674, 578, 907, 315, 865, 374, 8206, 88, 11, 779, 1121, 374, 1101, 8206, 88, 198, 14196, 19884, 1271, 3790, 5157, 1405, 8206, 1918, 323, 33032, 1918, 527, 11798, 22009, 4028, 15840, 15823, 11, 499, 649, 21650, 1817, 279, 12974, 3044, 13, 1789, 3187, 1473, 74694, 12958, 198, 87, 284, 220, 20, 198, 1407, 284, 1845, 2120, 8, 220, 674, 32430, 398, 34537, 865, 311, 264, 2777, 907, 198, 14196, 19884, 2028, 26420, 430, 279, 1121, 374, 2744, 264, 2777, 907, 11, 15851, 315, 279, 4221, 596, 8206, 1918, 323, 33032, 1918, 5718, 13, 128009]\n"
     ]
    }
   ],
   "source": [
    "space = tokenizer(\" \", add_special_tokens = False).input_ids[0]\n",
    "tokenizer.decode([space if x == -100 else x for x in trainer.train_dataset[5][\"labels\"]])\n",
    "print(trainer.train_dataset[0][\"labels\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3enWUM0jV-jV"
   },
   "source": [
    "我们可以看到系统和指令提示已成功屏蔽。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "cellView": "form",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:09:13.136253Z",
     "iopub.status.busy": "2024-12-30T09:09:13.135684Z",
     "iopub.status.idle": "2024-12-30T09:09:13.141100Z",
     "shell.execute_reply": "2024-12-30T09:09:13.140345Z",
     "shell.execute_reply.started": "2024-12-30T09:09:13.136229Z"
    },
    "id": "2ejIt2xSNKKp",
    "outputId": "ac07343f-67db-44e4-f9d3-83539724e6af",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPU = NVIDIA A100-SXM4-80GB. 最大内存 = 79.347 GB.\n",
      "2.635 GB 的内存已保留。\n"
     ]
    }
   ],
   "source": [
    "#@title 显示当前内存状态\n",
    "gpu_stats = torch.cuda.get_device_properties(0)\n",
    "start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
    "max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\n",
    "print(f\"GPU = {gpu_stats.name}. 最大内存 = {max_memory} GB.\")\n",
    "print(f\"{start_gpu_memory} GB 的内存已保留。\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:09:15.340607Z",
     "iopub.status.busy": "2024-12-30T09:09:15.340272Z",
     "iopub.status.idle": "2024-12-30T09:10:56.381821Z",
     "shell.execute_reply": "2024-12-30T09:10:56.381226Z",
     "shell.execute_reply.started": "2024-12-30T09:09:15.340583Z"
    },
    "id": "yqxqAZ7KJ4oL",
    "outputId": "fb3dc2a2-5cd6-4aa0-dfc5-ad734359f397",
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "==((====))==  Unsloth - 2x faster free finetuning | Num GPUs = 1\n",
      "   \\\\   /|    Num examples = 100,000 | Num Epochs = 1\n",
      "O^O/ \\_/ \\    Batch size per device = 2 | Gradient Accumulation steps = 4\n",
      "\\        /    Total batch size = 8 | Total steps = 60\n",
      " \"-____-\"     Number of trainable parameters = 24,313,856\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='60' max='60' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [60/60 01:26, Epoch 0/1]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.793500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.839500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.091000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.904500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.779700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.924900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.624100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.003900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.856700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.752200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.885600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.091500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.936600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.640300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>0.870000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>0.632300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>1.001600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>0.829400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>0.765800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>0.923300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21</td>\n",
       "      <td>0.893800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>22</td>\n",
       "      <td>0.849500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>23</td>\n",
       "      <td>1.028500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>24</td>\n",
       "      <td>0.871800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25</td>\n",
       "      <td>0.637500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26</td>\n",
       "      <td>0.829400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27</td>\n",
       "      <td>0.830200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>28</td>\n",
       "      <td>0.783600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>29</td>\n",
       "      <td>1.083400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30</td>\n",
       "      <td>1.030400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31</td>\n",
       "      <td>0.711100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32</td>\n",
       "      <td>0.540800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33</td>\n",
       "      <td>0.657700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>34</td>\n",
       "      <td>0.574700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>35</td>\n",
       "      <td>0.763000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>36</td>\n",
       "      <td>0.995500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>37</td>\n",
       "      <td>0.897600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>38</td>\n",
       "      <td>0.713900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>39</td>\n",
       "      <td>0.779500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>40</td>\n",
       "      <td>0.998500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>41</td>\n",
       "      <td>0.740600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>42</td>\n",
       "      <td>0.998300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>43</td>\n",
       "      <td>0.771500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>44</td>\n",
       "      <td>0.810500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>45</td>\n",
       "      <td>0.762400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>46</td>\n",
       "      <td>0.862600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>47</td>\n",
       "      <td>0.788200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>48</td>\n",
       "      <td>0.647500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>49</td>\n",
       "      <td>1.020300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>1.028800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>51</td>\n",
       "      <td>0.456000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>52</td>\n",
       "      <td>0.908400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>53</td>\n",
       "      <td>1.298500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>54</td>\n",
       "      <td>0.690500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>55</td>\n",
       "      <td>1.059200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>56</td>\n",
       "      <td>1.115600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>57</td>\n",
       "      <td>0.721900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>58</td>\n",
       "      <td>0.828800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>59</td>\n",
       "      <td>0.753700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>60</td>\n",
       "      <td>0.915500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "trainer_stats = trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "cellView": "form",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:11:00.532807Z",
     "iopub.status.busy": "2024-12-30T09:11:00.532546Z",
     "iopub.status.idle": "2024-12-30T09:11:00.537648Z",
     "shell.execute_reply": "2024-12-30T09:11:00.537064Z",
     "shell.execute_reply.started": "2024-12-30T09:11:00.532788Z"
    },
    "id": "pCqnaKmlO1U9",
    "outputId": "fcbecf7f-b8a1-45d5-f415-eec2bdf96576",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "96.2098 秒用于训练。\n",
      "1.6 分钟用于训练。\n",
      "峰值保留内存 = 3.66 GB。\n",
      "训练的峰值保留内存 = 1.025 GB。\n",
      "峰值保留内存占最大内存的百分比 = 4.613 %。\n",
      "训练的峰值保留内存占最大内存的百分比 = 1.292 %。\n"
     ]
    }
   ],
   "source": [
    "#@title 显示最终内存和时间统计\n",
    "used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
    "used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n",
    "used_percentage = round(used_memory         /max_memory*100, 3)\n",
    "lora_percentage = round(used_memory_for_lora/max_memory*100, 3)\n",
    "print(f\"{trainer_stats.metrics['train_runtime']} 秒用于训练。\")\n",
    "print(f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} 分钟用于训练。\")\n",
    "print(f\"峰值保留内存 = {used_memory} GB。\")\n",
    "print(f\"训练的峰值保留内存 = {used_memory_for_lora} GB。\")\n",
    "print(f\"峰值保留内存占最大内存的百分比 = {used_percentage} %。\")\n",
    "print(f\"训练的峰值保留内存占最大内存的百分比 = {lora_percentage} %。\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ekOmTR1hSNcr"
   },
   "source": [
    "<a name=\"Inference\"></a>\n",
    "### 推理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:11:04.389048Z",
     "iopub.status.busy": "2024-12-30T09:11:04.388515Z",
     "iopub.status.idle": "2024-12-30T09:11:05.337550Z",
     "shell.execute_reply": "2024-12-30T09:11:05.337059Z",
     "shell.execute_reply.started": "2024-12-30T09:11:04.389022Z"
    },
    "id": "kR3gIAX-SM2q",
    "outputId": "53188d07-ba68-420e-874b-1bace9929aa0",
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nCutting Knowledge Date: December 2023\\nToday Date: 26 July 2024\\n\\n<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n继续斐波那契数列：1, 1, 2, 3, 5, 8,<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\nContinue the Fibonacci sequence:\\n\\n9, 13, 21, 34, 55, 89<|eot_id|>']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from unsloth.chat_templates import get_chat_template\n",
    "\n",
    "tokenizer = get_chat_template(\n",
    "    tokenizer,\n",
    "    chat_template = \"llama-3.1\",\n",
    ")\n",
    "FastLanguageModel.for_inference(model) # 启用原生2倍速推理\n",
    "\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": \"继续斐波那契数列：1, 1, 2, 3, 5, 8,\"},\n",
    "]\n",
    "inputs = tokenizer.apply_chat_template(\n",
    "    messages,\n",
    "    tokenize = True,\n",
    "    add_generation_prompt = True, # 必须添加以进行生成\n",
    "    return_tensors = \"pt\",\n",
    ").to(\"cuda\")\n",
    "\n",
    "outputs = model.generate(input_ids = inputs, max_new_tokens = 64, use_cache = True,\n",
    "                         temperature = 1.5, min_p = 0.1)\n",
    "tokenizer.batch_decode(outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CrSvZObor0lY"
   },
   "source": [
    " 可以使用 `TextStreamer` 进行流式推理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:11:09.525980Z",
     "iopub.status.busy": "2024-12-30T09:11:09.524949Z",
     "iopub.status.idle": "2024-12-30T09:11:14.338719Z",
     "shell.execute_reply": "2024-12-30T09:11:14.338199Z",
     "shell.execute_reply.started": "2024-12-30T09:11:09.525923Z"
    },
    "id": "e2pEuRb1r2Vg",
    "outputId": "d4096dc5-c359-49c0-c08f-f8c890e414e5",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "继续斐波那契数列：13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765, 10946, 17711, 28657, 46368, 75025, 121393, 196418, 317811, 514229, 832040, 1346269, 2178309, 3524578, 5702887, 9227465, 14930352, 24157817\n"
     ]
    }
   ],
   "source": [
    "FastLanguageModel.for_inference(model) # 启用本地2倍速度推理\n",
    "\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": \"继续斐波那契数列：1, 1, 2, 3, 5, 8,\"},\n",
    "]\n",
    "inputs = tokenizer.apply_chat_template(\n",
    "    messages,\n",
    "    tokenize = True,\n",
    "    add_generation_prompt = True, # 必须添加以进行生成\n",
    "    return_tensors = \"pt\",\n",
    ").to(\"cuda\")\n",
    "\n",
    "from transformers import TextStreamer\n",
    "text_streamer = TextStreamer(tokenizer, skip_prompt = True)\n",
    "_ = model.generate(input_ids = inputs, streamer = text_streamer, max_new_tokens = 128,\n",
    "                   use_cache = True, temperature = 1.5, min_p = 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uMuVrWbjAzhc"
   },
   "source": [
    "<a name=\"Save\"></a>\n",
    "### 保存和加载微调模型\n",
    "要将最终模型保存为LoRA适配器，可以使用Huggingface的`push_to_hub`进行在线保存，或使用`save_pretrained`进行本地保存。\n",
    "\n",
    "**[注意]** 这只保存LoRA适配器，而不是完整模型。要保存为16位或GGUF格式，请向下滚动！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:11:18.624478Z",
     "iopub.status.busy": "2024-12-30T09:11:18.623826Z",
     "iopub.status.idle": "2024-12-30T09:11:19.221555Z",
     "shell.execute_reply": "2024-12-30T09:11:19.220989Z",
     "shell.execute_reply.started": "2024-12-30T09:11:18.624456Z"
    },
    "id": "upcOlWe7A1vc",
    "outputId": "c962e43a-2027-4a39-ea03-870b707a22d4",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('lora_model/tokenizer_config.json',\n",
       " 'lora_model/special_tokens_map.json',\n",
       " 'lora_model/tokenizer.json')"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.save_pretrained(\"lora_model\") # 本地保存\n",
    "tokenizer.save_pretrained(\"lora_model\")\n",
    "# model.push_to_hub(\"your_name/lora_model\", token = \"...\") # 在线保存\n",
    "# tokenizer.push_to_hub(\"your_name/lora_model\", token = \"...\") # 在线保存"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AEEcJ4qfC7Lp"
   },
   "source": [
    "现在，如果你想加载我们刚刚保存的LoRA适配器进行推理，请将`False`设置为`True`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:11:28.724525Z",
     "iopub.status.busy": "2024-12-30T09:11:28.723994Z",
     "iopub.status.idle": "2024-12-30T09:11:29.128157Z",
     "shell.execute_reply": "2024-12-30T09:11:29.127705Z",
     "shell.execute_reply.started": "2024-12-30T09:11:28.724492Z"
    },
    "id": "MKX_XKs_BNZR",
    "outputId": "f22dbd75-ea37-48bb-9f75-4178aebe9353",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "答案：埃夫兰塔高塔。<|eot_id|>\n"
     ]
    }
   ],
   "source": [
    "if False:\n",
    "    from unsloth import FastLanguageModel\n",
    "    model, tokenizer = FastLanguageModel.from_pretrained(\n",
    "        model_name = \"lora_model\", # 你用于训练的模型\n",
    "        max_seq_length = max_seq_length,\n",
    "        dtype = dtype,\n",
    "        load_in_4bit = load_in_4bit,\n",
    "    )\n",
    "    FastLanguageModel.for_inference(model) # 启用本地2倍速推理\n",
    "\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": \"描述法国首都的一座高塔。\"},\n",
    "]\n",
    "inputs = tokenizer.apply_chat_template(\n",
    "    messages,\n",
    "    tokenize = True,\n",
    "    add_generation_prompt = True, # 必须添加用于生成\n",
    "    return_tensors = \"pt\",\n",
    ").to(\"cuda\")\n",
    "\n",
    "from transformers import TextStreamer\n",
    "text_streamer = TextStreamer(tokenizer, skip_prompt = True)\n",
    "_ = model.generate(input_ids = inputs, streamer = text_streamer, max_new_tokens = 128,\n",
    "                   use_cache = True, temperature = 1.5, min_p = 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f422JgM9sdVT"
   },
   "source": [
    "### 保存为 VLLM 的 float16\n",
    "\n",
    "也支持直接保存为 `float16`。选择 `merged_16bit` 以保存为 float16 或选择 `merged_4bit` 以保存为 int4。我们还允许使用 `lora` 适配器作为后备选项"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecutionIndicator": {
     "show": true
    },
    "execution": {
     "iopub.execute_input": "2024-12-30T09:11:43.046388Z",
     "iopub.status.busy": "2024-12-30T09:11:43.046125Z",
     "iopub.status.idle": "2024-12-30T09:12:07.432529Z",
     "shell.execute_reply": "2024-12-30T09:12:07.431990Z",
     "shell.execute_reply.started": "2024-12-30T09:11:43.046369Z"
    },
    "id": "iHjt_SMYsd3P",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unsloth: Merging 4bit and LoRA weights to 16bit...\n",
      "Unsloth: Will use up to 142.73 out of 200.0 RAM for saving.\n",
      "Unsloth: Saving model... This might take 5 minutes ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 28/28 [00:00<00:00, 116.78it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unsloth: Saving tokenizer... Done.\n",
      "Done.\n"
     ]
    }
   ],
   "source": [
    "# 合并为16位\n",
    "if False: model.save_pretrained_merged(\"model\", tokenizer, save_method = \"merged_16bit\",)\n",
    "\n",
    "# 合并为4位\n",
    "if False: model.save_pretrained_merged(\"model\", tokenizer, save_method = \"merged_4bit\",)\n",
    "\n",
    "# 仅LoRA适配器\n",
    "if False: model.save_pretrained_merged(\"model\", tokenizer, save_method = \"lora\",)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TCv4vXHd61i7"
   },
   "source": [
    "### GGUF / llama.cpp 转换\n",
    "Unsloth支持原生保存到 `GGUF` / `llama.cpp`。使用 `save_pretrained_gguf` 进行本地保存。\n",
    "\n",
    "支持的部分量化方法如下：\n",
    "* `q8_0` - 快速转换。高资源使用，但通常可以接受。\n",
    "* `q4_k_m` - 推荐。对一半的 attention.wv 和 feed_forward.w2 张量使用 Q6_K，其他使用 Q4_K。\n",
    "* `q5_k_m` - 推荐。对一半的 attention.wv 和 feed_forward.w2 张量使用 Q6_K，其他使用 Q5_K。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FqfebeAdT073",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 保存为8bit Q8_0\n",
    "if False: model.save_pretrained_gguf(\"model\", tokenizer,)\n",
    "\n",
    "# 保存为16bit GGUF\n",
    "if False: model.save_pretrained_gguf(\"model\", tokenizer, quantization_method = \"f16\")\n",
    "\n",
    "# 保存为q4_k_m GGUF\n",
    "if False: model.save_pretrained_gguf(\"model\", tokenizer, quantization_method = \"q4_k_m\")"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "007a35a241b346ec9a5cdd6f3e4ddd27": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4083b2ef8e6348e18b69d116508b46ff",
      "placeholder": "​",
      "style": "IPY_MODEL_9555be409a2c4a97b18d4978ed13d35f",
      "value": "README.md: 100%"
     }
    },
    "098bd8ace574423da763eb0eae1d3bb6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "0b8cf23562ae4e428ab4f00979e7ce07": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0bfbfe620ff446a0a47f7d5de7c88975": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0c30ded692064dc7bf36a93897f2b68f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0d384a63bb5f4b7384ac5f20d1a399b0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "0faf73ed502649c4855adca930e3c8ff": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "18c4cb423f8744afa2b85ee361dcc121": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_76fb012b818f4bfb88c103262fb9c20e",
       "IPY_MODEL_8158ce20191d40fba140ed2c2f4ed859",
       "IPY_MODEL_595694711ab148e8bbf9824896c10468"
      ],
      "layout": "IPY_MODEL_4b355eddebb548a7b60d28c12403ebfd"
     }
    },
    "1c0c2835705f41089de4caea98127c04": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1e4ea03959b3496f8e75cc3588cf347c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "20e75852859845f697760003945b952f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4a91a45aba634535a53a49a873b25dfe",
      "placeholder": "​",
      "style": "IPY_MODEL_d773266178564b6cb739d756ff8de459",
      "value": "model.safetensors: 100%"
     }
    },
    "23907906314743938db4e484c15480cc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "252c3d25fe3341379e5fc31d30dee7bc": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "254473850de1471e9584a9c03962bd6f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_56e59a69f84e487f91e263313f5d2b72",
       "IPY_MODEL_524b13439d53402eb35388606058c288",
       "IPY_MODEL_fc7d250163bb470fa19503356dfe9e81"
      ],
      "layout": "IPY_MODEL_9dc2697fe6984649b0ac36018f6904c9"
     }
    },
    "2a6ca29a76ff430d86213f910858db5b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_95249b8fb6a84054a01f22c5f73f207b",
      "placeholder": "​",
      "style": "IPY_MODEL_2ed2b017b9a24f36a4222c5c27753991",
      "value": " 100000/100000 [00:01&lt;00:00, 63603.83 examples/s]"
     }
    },
    "2aea4e74b6c4449ea24812d11675daee": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "2b359412d4914aa38a6e21284c12ecbc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ddee625828cb4c22927aa73a02cd2dd9",
      "placeholder": "​",
      "style": "IPY_MODEL_fd46f381983f49179de05497c171c805",
      "value": " 117M/117M [00:00&lt;00:00, 210MB/s]"
     }
    },
    "2bb75539976c49ed805c4ff6c58fb1d2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2ed2b017b9a24f36a4222c5c27753991": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "35aee367658940b9b2f6884ea87b3e4b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "3601b111b63f401cb5c75ba08ae74b61": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d8c3cd1d08b747b690a59b7793aaf742",
      "placeholder": "​",
      "style": "IPY_MODEL_95a6cb0e04984f3c96bbe9ee9c5fa287",
      "value": " 9.09M/9.09M [00:00&lt;00:00, 34.6MB/s]"
     }
    },
    "378176d2f0c9466d8762a584edf4217d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "39684b70f2ff48cab454617c721f7777": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_e8445e90b1054aacbecf198c7979a0b6",
       "IPY_MODEL_d1cc50fb6d5849888af5d765dc51ab62",
       "IPY_MODEL_2b359412d4914aa38a6e21284c12ecbc"
      ],
      "layout": "IPY_MODEL_a4ceb6dbc8de4fa798ee39d28e5ebc40"
     }
    },
    "39bf1c29894f43acb6d2919e64a4fd28": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_007a35a241b346ec9a5cdd6f3e4ddd27",
       "IPY_MODEL_969a119573f942b29951ae2933e61cde",
       "IPY_MODEL_b8c4d378ea0e4bcd9f572a191a7c136f"
      ],
      "layout": "IPY_MODEL_7d37dd0e06724b53b4f31cc0a4321b0d"
     }
    },
    "3cd95b7c5e2f4c6883333045db11c6d6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3ea24021851a4341b3cb30d4ff0d495f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3ebff50e3c0b4cb4b99c93adde35b712": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3ffe42931dcf4a69972f4d50ee4dd3dd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ee9dcec2d5c44fd883f16c06b9f76264",
       "IPY_MODEL_982b6b94642d49fa85fab6ad621392fe",
       "IPY_MODEL_42990f347a8c42f7b510e2d17c7d3c6e"
      ],
      "layout": "IPY_MODEL_3cd95b7c5e2f4c6883333045db11c6d6"
     }
    },
    "4083b2ef8e6348e18b69d116508b46ff": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "42990f347a8c42f7b510e2d17c7d3c6e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_74dc78a38e30465a96d2c8a22a27b127",
      "placeholder": "​",
      "style": "IPY_MODEL_c6b4759ce826421081508270cb30334b",
      "value": " 100000/100000 [03:00&lt;00:00, 544.59 examples/s]"
     }
    },
    "45bc9d882a8f4a7e813245b1590d4427": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "4606a430f0214a0bb524a06ab78af1a9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_700b2fc6799c41f4984e07c36502f2c9",
       "IPY_MODEL_be744decce12471388b689a085576c7f",
       "IPY_MODEL_e8efa5951c4b4f3e9922e029c57a5ab3"
      ],
      "layout": "IPY_MODEL_afa86659651d48a7b4a3043a0909322f"
     }
    },
    "4a91a45aba634535a53a49a873b25dfe": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4b355eddebb548a7b60d28c12403ebfd": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4b41aa65c6894e918b04709f8e9270d2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4f0aaa3ce8ca43fdaea4cd4f7b595218": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6d163ef8e07b4a1cba351ec44175efd5",
      "placeholder": "​",
      "style": "IPY_MODEL_9a4ea589183346dab240125cd38ba0f9",
      "value": "tokenizer.json: 100%"
     }
    },
    "524b13439d53402eb35388606058c288": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f7f53b3a53eb4b8e9fa47cd233f4df7f",
      "max": 454,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_73efb213f4a84996a353495b912393d9",
      "value": 454
     }
    },
    "5628ed38f304438faf5442b29a9511d6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "56e59a69f84e487f91e263313f5d2b72": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3ebff50e3c0b4cb4b99c93adde35b712",
      "placeholder": "​",
      "style": "IPY_MODEL_35aee367658940b9b2f6884ea87b3e4b",
      "value": "special_tokens_map.json: 100%"
     }
    },
    "58ce4633471c438db6e103a1ca3806a0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "595694711ab148e8bbf9824896c10468": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0b8cf23562ae4e428ab4f00979e7ce07",
      "placeholder": "​",
      "style": "IPY_MODEL_b774631435d4450da64f0a2b40cbf3aa",
      "value": " 54.6k/54.6k [00:00&lt;00:00, 3.20MB/s]"
     }
    },
    "5b34a4e8fc7747e78b49ad5bf67a6580": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5c9ee920068a47d89dbf5cbdd9e848a3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "5e9825466cd2481b92cfe89f33b11fe3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c685f29a5d2c461ca3dda867bab6df50",
      "placeholder": "​",
      "style": "IPY_MODEL_e2f16d56b21c4ff2918872d70e5ca847",
      "value": "Generating train split: 100%"
     }
    },
    "6064feeea79040409e18a1e2a289b09a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_bb241a26ca4d4d7186ba46cda1f8a802",
       "IPY_MODEL_c9abb42da1734388a7d2f1a06832ecc6",
       "IPY_MODEL_7c3a37494e5848b9994b37a4c8bac132"
      ],
      "layout": "IPY_MODEL_c668ae4c7d174f2dad3fb837ff873e57"
     }
    },
    "672262ede93c413cbc219ca4fec5b98a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "683dd21b9eef4bb4a0c9bc15c984703a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "6d163ef8e07b4a1cba351ec44175efd5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6d52daf29c90402a9762acdde765713f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "6e0fe945001140b3959e617a2f55c353": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "700b2fc6799c41f4984e07c36502f2c9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9edc5f3740e54d7ab6fd38f270f4a40e",
      "placeholder": "​",
      "style": "IPY_MODEL_2aea4e74b6c4449ea24812d11675daee",
      "value": "generation_config.json: 100%"
     }
    },
    "7028f81557e54d0d8df4cc1a0f85b12c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "70b0793d26c14afbbe8af319d38dbb07": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_20e75852859845f697760003945b952f",
       "IPY_MODEL_f0c7c3402e6549148ac293dcf01ae8da",
       "IPY_MODEL_b1c8acb40b924d8f854408252c1e061c"
      ],
      "layout": "IPY_MODEL_aa28da40d8074e89ad32b5cb5ad04fa6"
     }
    },
    "73efb213f4a84996a353495b912393d9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "74dc78a38e30465a96d2c8a22a27b127": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "757a4328afef4293b6d31ce9b90eec79": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "76e2e47c93e541ff820bcbab9264381d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "76fb012b818f4bfb88c103262fb9c20e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0faf73ed502649c4855adca930e3c8ff",
      "placeholder": "​",
      "style": "IPY_MODEL_b0670b92e79744c5ba11501bafdc101b",
      "value": "tokenizer_config.json: 100%"
     }
    },
    "785d9147f4a341afafc5c5743892df16": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_5e9825466cd2481b92cfe89f33b11fe3",
       "IPY_MODEL_bfbb37b6f4b247b5bf5aaf7e1d80bcf9",
       "IPY_MODEL_2a6ca29a76ff430d86213f910858db5b"
      ],
      "layout": "IPY_MODEL_92d981a21b204f6c8b52e3caa16d1784"
     }
    },
    "7c3a37494e5848b9994b37a4c8bac132": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d5cfa138483f4007b2a95be833043235",
      "placeholder": "​",
      "style": "IPY_MODEL_6d52daf29c90402a9762acdde765713f",
      "value": " 100000/100000 [01:07&lt;00:00, 2101.01 examples/s]"
     }
    },
    "7d37dd0e06724b53b4f31cc0a4321b0d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8158ce20191d40fba140ed2c2f4ed859": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3ea24021851a4341b3cb30d4ff0d495f",
      "max": 54598,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_7028f81557e54d0d8df4cc1a0f85b12c",
      "value": 54598
     }
    },
    "8578be7c3f4044d3bfe0eae94fe2c21a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_4f0aaa3ce8ca43fdaea4cd4f7b595218",
       "IPY_MODEL_8611f4c6d12641dd960377346daba5d1",
       "IPY_MODEL_3601b111b63f401cb5c75ba08ae74b61"
      ],
      "layout": "IPY_MODEL_a5bde1dcf7f745faa140cb6728c415fc"
     }
    },
    "8611f4c6d12641dd960377346daba5d1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d8f739452aa5410aa5eba87b2760c2af",
      "max": 9085657,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_dc098c4a448243ddb82ab769dc261171",
      "value": 9085657
     }
    },
    "870ff8f17c7b47ec8d49cac84216b04c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "8b09015cc22240cc8e736f7050c0f37c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8c5ad85b4da14b239340ac95244d8ed4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "904e7bac43bd4333b321cacfed5dcb60": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "92d981a21b204f6c8b52e3caa16d1784": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "95249b8fb6a84054a01f22c5f73f207b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9555be409a2c4a97b18d4978ed13d35f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "95a6cb0e04984f3c96bbe9ee9c5fa287": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "969a119573f942b29951ae2933e61cde": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5628ed38f304438faf5442b29a9511d6",
      "max": 982,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_6e0fe945001140b3959e617a2f55c353",
      "value": 982
     }
    },
    "982b6b94642d49fa85fab6ad621392fe": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_378176d2f0c9466d8762a584edf4217d",
      "max": 100000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_e221482cbe95465191212d85d539938c",
      "value": 100000
     }
    },
    "9a4ea589183346dab240125cd38ba0f9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "9a8f1b8079fe478ebf0b16096cb224f5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9dc2697fe6984649b0ac36018f6904c9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9edc5f3740e54d7ab6fd38f270f4a40e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a326b2e89f1c46f28cd166afc7490e2b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_58ce4633471c438db6e103a1ca3806a0",
      "placeholder": "​",
      "style": "IPY_MODEL_cf1b769b7a744b5f8bccf6798566582f",
      "value": "Standardizing format: 100%"
     }
    },
    "a4ceb6dbc8de4fa798ee39d28e5ebc40": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a5bde1dcf7f745faa140cb6728c415fc": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "aa28da40d8074e89ad32b5cb5ad04fa6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ae2464c1cbc442a383de7577d2986116": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4b41aa65c6894e918b04709f8e9270d2",
      "placeholder": "​",
      "style": "IPY_MODEL_cdae06929214464ea25e343f17b4a843",
      "value": " 100000/100000 [00:20&lt;00:00, 7158.71 examples/s]"
     }
    },
    "afa86659651d48a7b4a3043a0909322f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b0670b92e79744c5ba11501bafdc101b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b1b0a4e3f00043b0a0eb7a053815a4a5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b1c8acb40b924d8f854408252c1e061c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f0377d014b2e4e89b9932750f04a6a3c",
      "placeholder": "​",
      "style": "IPY_MODEL_0d384a63bb5f4b7384ac5f20d1a399b0",
      "value": " 2.24G/2.24G [00:18&lt;00:00, 400MB/s]"
     }
    },
    "b774631435d4450da64f0a2b40cbf3aa": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b8c4d378ea0e4bcd9f572a191a7c136f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0c30ded692064dc7bf36a93897f2b68f",
      "placeholder": "​",
      "style": "IPY_MODEL_8c5ad85b4da14b239340ac95244d8ed4",
      "value": " 982/982 [00:00&lt;00:00, 21.3kB/s]"
     }
    },
    "bb241a26ca4d4d7186ba46cda1f8a802": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_dd30f3ead6394317be5a72aa890adfb9",
      "placeholder": "​",
      "style": "IPY_MODEL_1e4ea03959b3496f8e75cc3588cf347c",
      "value": "Map: 100%"
     }
    },
    "bcf8e36d938a4d959c31ea4ff3c8d4cf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_cc0bd79ca9e847fba88aafe2d612ffe4",
      "max": 100000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_76e2e47c93e541ff820bcbab9264381d",
      "value": 100000
     }
    },
    "bd71b6cb29e147ab9b10d1b85908c413": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c03b9410af384397849ef63b62f2c689",
      "placeholder": "​",
      "style": "IPY_MODEL_098bd8ace574423da763eb0eae1d3bb6",
      "value": " 100000/100000 [00:08&lt;00:00, 16117.89 examples/s]"
     }
    },
    "be744decce12471388b689a085576c7f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_252c3d25fe3341379e5fc31d30dee7bc",
      "max": 184,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_672262ede93c413cbc219ca4fec5b98a",
      "value": 184
     }
    },
    "bfbb37b6f4b247b5bf5aaf7e1d80bcf9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0bfbfe620ff446a0a47f7d5de7c88975",
      "max": 100000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_5c9ee920068a47d89dbf5cbdd9e848a3",
      "value": 100000
     }
    },
    "c03b9410af384397849ef63b62f2c689": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c668ae4c7d174f2dad3fb837ff873e57": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c685f29a5d2c461ca3dda867bab6df50": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c6b4759ce826421081508270cb30334b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c7907c51d16a4aeb87974c36b3b9e7c8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c9abb42da1734388a7d2f1a06832ecc6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d356b597dda14c7ab023403ee6959cf8",
      "max": 100000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_870ff8f17c7b47ec8d49cac84216b04c",
      "value": 100000
     }
    },
    "cc0bd79ca9e847fba88aafe2d612ffe4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "cdae06929214464ea25e343f17b4a843": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "cf1b769b7a744b5f8bccf6798566582f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "cff4f20460a442839c26bfa468b35dee": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d08e764aa8b94e7f9e1c727b53980abe": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_e62f6eb58a744d38b837e47d8a16db67",
       "IPY_MODEL_bcf8e36d938a4d959c31ea4ff3c8d4cf",
       "IPY_MODEL_ae2464c1cbc442a383de7577d2986116"
      ],
      "layout": "IPY_MODEL_9a8f1b8079fe478ebf0b16096cb224f5"
     }
    },
    "d1cc50fb6d5849888af5d765dc51ab62": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "danger",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2bb75539976c49ed805c4ff6c58fb1d2",
      "max": 116531415,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_45bc9d882a8f4a7e813245b1590d4427",
      "value": 116531404
     }
    },
    "d356b597dda14c7ab023403ee6959cf8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d5cfa138483f4007b2a95be833043235": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d6ab4d4143ff49bcae30be1bc2d76762": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d773266178564b6cb739d756ff8de459": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d7e0024b98a94a9fa12dc4154ff2b2fc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d8c3cd1d08b747b690a59b7793aaf742": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d8f739452aa5410aa5eba87b2760c2af": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "dc098c4a448243ddb82ab769dc261171": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "dd30f3ead6394317be5a72aa890adfb9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "dd9e90f2c16541e8a72c6771c4685b9a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_a326b2e89f1c46f28cd166afc7490e2b",
       "IPY_MODEL_eb855a0fcb554a8eb245351b3593623d",
       "IPY_MODEL_bd71b6cb29e147ab9b10d1b85908c413"
      ],
      "layout": "IPY_MODEL_b1b0a4e3f00043b0a0eb7a053815a4a5"
     }
    },
    "ddee625828cb4c22927aa73a02cd2dd9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e221482cbe95465191212d85d539938c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "e2d886444f0047fa9e2245b9773ced9e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "e2f16d56b21c4ff2918872d70e5ca847": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e4bf3f8e63bb4c01bbe821d438445d91": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e62f6eb58a744d38b837e47d8a16db67": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e4bf3f8e63bb4c01bbe821d438445d91",
      "placeholder": "​",
      "style": "IPY_MODEL_d7e0024b98a94a9fa12dc4154ff2b2fc",
      "value": "Map: 100%"
     }
    },
    "e8445e90b1054aacbecf198c7979a0b6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d6ab4d4143ff49bcae30be1bc2d76762",
      "placeholder": "​",
      "style": "IPY_MODEL_904e7bac43bd4333b321cacfed5dcb60",
      "value": "train-00000-of-00001.parquet: 100%"
     }
    },
    "e8efa5951c4b4f3e9922e029c57a5ab3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ea0f4a924ce14b97a53ba2b045ba1b1d",
      "placeholder": "​",
      "style": "IPY_MODEL_757a4328afef4293b6d31ce9b90eec79",
      "value": " 184/184 [00:00&lt;00:00, 14.6kB/s]"
     }
    },
    "ea0f4a924ce14b97a53ba2b045ba1b1d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "eb855a0fcb554a8eb245351b3593623d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1c0c2835705f41089de4caea98127c04",
      "max": 100000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_e2d886444f0047fa9e2245b9773ced9e",
      "value": 100000
     }
    },
    "ee9dcec2d5c44fd883f16c06b9f76264": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5b34a4e8fc7747e78b49ad5bf67a6580",
      "placeholder": "​",
      "style": "IPY_MODEL_23907906314743938db4e484c15480cc",
      "value": "Map (num_proc=2): 100%"
     }
    },
    "f0377d014b2e4e89b9932750f04a6a3c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f0c7c3402e6549148ac293dcf01ae8da": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "danger",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_cff4f20460a442839c26bfa468b35dee",
      "max": 2242762780,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_683dd21b9eef4bb4a0c9bc15c984703a",
      "value": 2242762567
     }
    },
    "f7f53b3a53eb4b8e9fa47cd233f4df7f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fc7d250163bb470fa19503356dfe9e81": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8b09015cc22240cc8e736f7050c0f37c",
      "placeholder": "​",
      "style": "IPY_MODEL_c7907c51d16a4aeb87974c36b3b9e7c8",
      "value": " 454/454 [00:00&lt;00:00, 21.8kB/s]"
     }
    },
    "fd46f381983f49179de05497c171c805": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
